Check for duplicate files

As I said before... the devil is in the details 🙂

30,000 is a lot of files to have in one directory. From what you said I expected it to be large, which is why I opted for a find in the shell rather than the Finder, but even then there are limitations.

That said, since you're still getting errors, the simplest solution would be to break down the script into a loop that processes a subset of the files each time, something replacing the line:

set folder1Files to every file of folder1

with:

repeat with firstChar in {"a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w", "x", "y", "z"}

set folder1Files to (every file of folder1 whose name begins with firstChar)

...

(with a corresponding 'end repeat' at the end). This will break the list of 30,000 files into (hopefully) more manageable chunks based on the first character - you can change that to break on any parameters you like if you need to

Conceptually this is pretty straightforward - a little AppleScript, or maybe shell script and you're done.

As always with these kinds of questions, though, the devil is in the details. In this case, what - specifically - constitutes a match? are files with the same name considered a match? what if one is newer/older/larger/smaller than the other? are they still considered a match? which one should you keep? the newest? the biggest?

Once you clarify that, the rest should be easy.

The following script (minimally tested!) should do what you want. Copy the script into a new Script Editor document and run it (the delete command is actually a misnomer... it only moves the files to the trash so there's still a chance of recovery.

set folder1 to (choose folder with prompt "Please select the folder to be cleaned")

set folder2 to (choose folder with prompt "Please select the folder to compare")

-- fast way to get a list files in a directory

set fileList to do shell script "/usr/bin/find " & quoted form of POSIX path of folder2 & " -type f -exec basename {} \\;"

set fileList to paragraphs of fileList

tell application "Finder"

set folder1Files to every file of folder1

repeat with eachFile in folder1Files

set fName to name of eachFile

if fName is in fileList then

-- we have a filename match

set f1Size to size of eachFile as integer

set matchingf2File to do shell script "/usr/bin/find " & quoted form of POSIX path of folder2 & " -type f -name " & quoted form of fName & " -size " & f1Size & "c"

if matchingf2File is not "" then

-- we have a duplicate, so:

deleteeachFile

end if

end if

end repeat

end tell

It probably needs some explanation...

It starts off by prompting for two folders - the first should be the one that contains the flat directory that you want to clean up (folder1). The second should be the one with the hierarchal directories you want to search in (folder2).

set folder1 to (choose folder with prompt "Please select the folder to be cleaned")

set folder2 to (choose folder with prompt "Please select the folder to compare")

Then it uses a shell command find to get a listing of all the files in folder2. I do this because the Finder is notoriously slow in traversing large directory trees, so even though using the Finder would be simpler, it would be much slower.

set fileList to do shell script "/usr/bin/find " & quoted form of POSIX path of folder2 & " -type f -exec basename {} \\;"

set fileList to paragraphs of fileList

Now I iterate through the files in folder1, checking the name of the file against the cached list of files in folder2.

set folder1Files to every file of folder1

repeat with eachFile in folder1Files

set fName to name of eachFile

if fName is in fileList then

If there are no matches I move on to the next file, otherwise we at least have a file that has the same name, so we need to check its size.

set f1Size to size of eachFile as integer

set matchingf2File to do shell script "/usr/bin/find " & quoted form of POSIX path of folder2 & " -type f -name " & quoted form of fName & " -size " & f1Size & "c"

Here I use another shell trick. I first get the size of the current file. I then perform another find to find a file that has the same size as the current find. If I get back an empty list I know the file sizes are different, so I leave the file alone, but if the file sizes match I know it's safe to delete the file.

I know this may sound convoluted, but for a large directory tree, with a large number of files in folder1, it would be cumbersome/unwieldy to perform a full depth traversal of folder2 for every file, so I first cache the list of file names and just do a secondary search for those that have matching filenames.

Wrap your Finder tell block with:

with timeout of 43200 seconds -- 12 hours

tell application "Finder"

…

end tell

end timeout